AO-All 3  237  NAVAL  POSTORAOUATE  SCHOOL  MONTEREY  CA  P/O  0/2 

SOFTWARE  MAINTENANCE!  IMPROVEMENT  THROUOH  SETTER  DEVELOPMENT  ST — ETC(U) 
FES  02  N  F  SC  MNE I  DC  WIND 

UNCLASSIFIED  NPS-5A-02-002  NL 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BURLAU  Of  STANDARDS  1963-A 


82  04  08  006 


NAVAL  POSTGRADUATE  SCHOOL 
Monterey,  California 


Rear  Admiral  J.  J.  Ekelund  David  R.  Schrady 

Superintendent  Acting  Provost 


The  work  reported  herein  was  supported  by  the  Trident  Command  and 
Control  Systems  Maintenance  Agency, 


Reproduction  of  all  or  part  of  this  report  is  authorized. 


This  report  was  prepared  by: 


'NORMAN  F.  jCHNEIDEWINb 
Professor  of  Computer  Science 


Reviewed  by: 


Released  by: 


5SL  /*<■ 


WILLIAM  M.  TOLLES 
Dean  of  Research 


Unclassified _ 

security  classification  of  this  page  ibam  Dm*  cm« •« 


REPORT  DOCUMENTATION  PAGE 


I  REPORT  NUMB 

NPS-54-82-002 


•-  TITLE  CmPSu *«»/.) 


READ  INSTRUCTIONS 
BEFORE  COMPLETINO  FORM 


1.  RECIPIENT'S  CATALOG  NUMBER 


s.  type  of  report  a  perioo  covered 

Final  Report 

SOFTWARE  MAINTENANCE:  IMPROVEMENT  THROUGH  1  Jam  80  to  1  Jan  82 

BETTER  DEVELOPMENT  STANDARDS  AND  DOCUMENTATION  TTenfonS^OOR 5T  report  number 


7.  author^; 


Norman  F.  Schneidewind 


»-  PERFORMING  ORGANIZATION  NAME  ANO  AOORESS 

Naval  Postgraduate  School 
Monterey,  California  93940 


II.  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Trident  Command  and  Control  Systems 
Maintenance  Agency 
Newport ,  Rhode  Island 


li 


CONTRACT  OR  grant  HUMBERT*; 


10.  program  element,  project,  task 

AREA  A  WORK  UNIT  NUMBERS 


N4216680WR00007 


12.  REPORT  DATE 

22  February  1982 


is.  HUM 

41 


MONITORING  AGENCY  NAME  A  AODRESSflf  <M  If  trait  <  tram  Canlralltnt  Olllca >  IS.  SECURITY  CLASS,  (at  Mi  i 

Unclassified 


IS.  DISTRIBUTION  STATEMENT  (ot  fAla  Rapsrij 

Approved  for  public  release:  distribution  unlimited. 


It.  KEY  WORDS  (Cmttimtm  m  rtvwM  uidm  tt  ahiimt  aW  tStmtlfy  fef  M«dt 


Software  maintenance 
Software  maintainability 
Software  standards 


Military  Standard  MIL-STD* 1679 
Weapons  Specification  WS  8506 
Traceability 


20.  ABSTRACT  rCiMMw  an  ittwaa  a/*a  ||  nacaaaarr  IMmliy  A?  AIM*  iwAaaj  » 

Software  maintenance  is  frequently  the  most  expensive  phase  Of  the  software 
life  cycle.  It  is  also  the  phase  which  has  received  insufficient  attention  by 
management  and  software  developers.  Software  standards  have  improved  the  ability 
of  the  software  community  to  develop  and  design  software.  Unfortunately,  most 
standards  do  not  deal  with  the  maintenance  phase  in  a  substantive  way.  Since 
maintainability  has  to  be  designed  into  the  software  and  cannot  be  achieved 
after  the  software  is  delivered,  it  is  necessary  to  have  software  standards 
which  explicitly  incorporate  requirements  for 


00  1  JAN  71  1473  F1  EDITION  OF  I  NOV  «l  if  OBSOLETE 
S/N  OlOl-OM-SSOl  • 


4-  •  fTT*?  I 


security  CLASSIFICATIOM  OF  THIS  PAOE  | 


TABLE  OF  CONTENTS 


Page  No. 


BACKGROUND  -  2 

I.  INTRODUCTION  -  4 

II.  OBJECTIVES  -  6 

III.  APPROACH  -  7 

IV.  GOOD  MAINTENANCE  PRACTICES  -  8 

A.  Design  Approaches  for  Achieving  Good  - - - -  8 

Maintainabi lity 

B.  Specification  and  Documentation  Requirements  - II 

for  Achieving  Good  Maintainability 

C.  Testing  Approaches  for  Achieving  Good  - - -  15 

Maintainability 

V.  EVALUATION  OF  WEAPONS  SPECIFICATION  WS  8506  19 

VI.  EVALUATION  OF  MILITARY  STANDARD  MIL- STD  1679  27 

DISCLAIMER  -  36 

DISTRIBUTION  LIST  -  37 


1 


BACKGROUND 


This  report  addresses  the  maintenance  phase  of  the  software  life  cycle 
and  the  methodologies  and  procedures  which,  if  employed  during  the  soft¬ 
ware  development  and  design  phases,  will  improve  software  maintainability. 
We  define  maintenance  as  that  activity  which  is  concerned  with  making 
changes  to  software  for  the  purpose  of  improving  or  correcting  the  soft¬ 
ware.  Maintainability  is  a  property  of  the  software  which  makes  the 
maintenance  activity  easy  to  perform,  i.e.,  changes  to  the  software  are 
easy  to  incorporate  and  do  not  lead  to  new  errors  in  the  software. 

The  Trident  Command  and  Control  Systems  Maintenance  Agency  (CCSMA) 
tasked  the  Naval  Postgraduate  School  to  study  and  evaluate  various  Navy 
software  standards  with  regard  to  their  applicability  to  software 
maintenance .  The  question  posed  was:  Could  these  standards,  accompanied 
by  basic  program  documentation,  such  as  a  listing,  provide  adequate 
guidance  for  a  new  programmer  to  maintain  software,  such  as  that  found 
in  the  Trident  Command  and  Control  Subsystem?  Related  to  this  question 
is  the  critical  relationship  between  development  and  maintenance.  For 
the  most  part,  the  standards  and  documentation  techniques  which  were 
reviewed  (e.g.,  MIL-STD  1679)  were  designed  to  be  used  for  software 
development  and  not  for  maintenance,  specifically.  This  situation  illus¬ 
trates  the  fundamental  problem  in  software  maintenance:  inadequate 
attention  to  this  important  and  high  cost  phase  of  the  software  life 
cycle.  Since  the  conditions  for  future  maintenance  problems  are  created 
during  development  and  design,  the  approach  taken  in  this  study  was  to 
look  to  these  phases  of  the  life  cycle  as  the  areas  having  the  greatest 
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potential  for  improving  the  maintainability  of  software.  This  concept 
suggests  that  maintainability  must  be  designed  into  the  software  and 
that  development  methodology  has  a  more  significant  effect  on  main¬ 
tainability  than  maintenance  practices  themselves.  However,  pre¬ 
maintenance  phase  activities  should  not  be  emphasized  to  the  exclusion 
of  maintenance  phase  activities.  Maintenance  practices,  such  as  patch¬ 
ing  programs  and  deficiencies  in  testing  subsequent  to  a  change  (e.g., 
lack  of  regression  testing) ,  obviously  exert  a  deleterious  effect  on 
the  quality  of  software.  Unfortunately  improvements  in  maintenance 
practices  cannot  cure  underlying  problems  which  are  introduced  into  the 
software  when  it  is  specified  and  designed.  Hence  it  is  clear  that 
significant  gains  in  maintainability  can  only  be  achieved  by  recog¬ 
nizing  maintainability  objectives  as  an  integral  part  of  development 
and  design. 

Consistent  with  the  above  problem  approach,  this  report  begins  with 
a  discussion  of  the  findings  of  a  number  of  researchers,  in  the  areas  of 
software  development  and  maintenance  practices ,  which  bear  on  the  problem 
of  improving  software  maintainability.  This  is  followed  by  an  evalua¬ 
tion  of  Navy  weapon  system  software  standards  (WS  8506  and  MIL-STD  1679) , 
from  the  standpoint  of  their  effectiveness  as  aids  to  the  maintenance 
programmer. 
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I.  INTRODUCTION 


Software  maintenance  is  one  of  the  most  expensive  phases  of  the 
system  life  cycle.  One  author  suggests  it  is  as  high  as  67  per  cent  of 
the  effort  in  large-scale  systems  [l] .  Despite  this  fact,  maintenance 
requirements  receive  little  attention  during  system  development.  There 
are  many  reasons  for  this  situation,  but  three  stand  out: 

1.  Maintenance  is  considered  less  glamorous,  interesting  and 
challenging  as  compared  to  system  design  and  programming;  hence,  there 
is  little  incentive  for  computer  personnel  to  become  involved  in  main¬ 
tenance  activities. 

2.  During  development  it  is  often  too  early  in  the  project  to 
foresee  problems  which  may  occur  during  maintenance;  as  a  consequence 
maintainability  is  not  provided  in  the  system  design. 

3.  Project  management  does  not  always  recognize  that  maintainabil¬ 
ity  considerations  should  be  an  integral  part  of  the  design  process. 

One  result  of  this  lack  of  attention  to  maintainability  require¬ 
ments  during  design  is  loss  of  treaceability.  This  is  defined  as  the 
ability  to  identify  the  technical  information  which  pertains  to  a 
software  error  which  has  been  detected  during  the  maintenance  phase 
and  thereby  trace  the  error  to  the  applicable  design  specifications 
and  user  requirements  statements.  The  same  traceability  capability 
is  also  required  for  software  improvements  which  are  made  during  the 
maintenance  phase.  It  is  clear  that  if  significant  improvements  are 
to  be  made  in  software  maintainability,  requirements  for  maintenance 


must  be  made  an  integral  part  of  the  software  specification  develop¬ 
ment  and  design  process.  In  fact,  as  mentioned  later,  some  software 
developers  reconmend  that  software  design  be  oriented  around  the  need  to 
maintain  the  software  in  its  operating  environment. 


II.  OBJECTIVES 


The  objectives  of  this  report  are: 

1.  In  particular,  to  evaluate  Weapons  Specification  (WS)  8506  [2] 
and  Military  Standard  (MIL- STD)  1679  [3]  with  regard  to  their  suitability 
for  computer  program  maintenance  in  embedded  computer  systems. 

2.  In  general,  to  discuss  and  propose  software  development  prac¬ 
tices  which  will  lead  to  improved  maintainability  for  any  software 
system. 

In  1.  emphasis  will  be  on  the  evaluation  of  the  ability  of  these 
documents  to  provide  for  traceability  during  program  maintenance. 
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III.  APPROACH 


The  WS  8506  and  MIL-STD  1679  documents  were  examined  with  regard  to 
their  effectiveness  for  software  maintenance  purposes — in  particular, 
their  capability  for  providing  traceability.  In  addition  to  this  specific 
aspect,  all  sections  of  these  documents  which  are  applicable  to  maintenance 
were  reviewed,  and  strong  and  weak  points  were  noted.  A  major  approach 
in  this  review  was  the  use  of  a  rich  body  of  literature  in  software 
engineering  which  was  employed  as  an  information  source  for  suggesting 
improvements  in  the  documents.  This  literature  and  our  concept  of  the 
characteristics  of  good  specifications  and  standards  for  maintenance 
were  used  to  develop  a  set  of  criteria  for  judging  WS  8506  and  MIL-STD 
1679. 
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IV.  GOOD  MAINTENANCE  PRACTICES 


A.  Design  Approaches  for  Achieving  Good  Maintainability 

As  described  in  Your don  [4],  a  good  software  design  and  maintain¬ 
ability  strategy  is  that  of  divide  and  conquer.  The  cost  of  implemen¬ 
tation  will  be  minimized  when  parts  of  the  problem  are  manageably  small 
and  separately  solvable.  Similarly,  the  cost  of  maintenance  will  be 
minimized  when  parts  of  the  system  are  easily  related  to  the  application 
(a  facet  of  traceabilty) ,  manageably  small  and  separately  correctable. 
This  modularity  concept  should  be  incorporated  in  weapons  systems 
software  specifications  and  standards .  Furthermore,  pieces  of  the 
problem  should  be  independent.  Highly  interrelated  parts  of  the  problem 
should  be  in  the  same  piece  of  the  system  and  unrelated  parts  should 
reside  in  unrelated  pieces  of  the  system.  Finally,  implementation  and 
maintenance  will  be  minimized  when  each  relationship  between  pieces  of 
the  system  corresponds  only  to  a  relationship  between  pieces  of  the 
problem. 

Of  great  interest  for  maintenance  is  the  fact  that  it  is  very 
difficult  to  simplify  the  structure  of  a  program  after  the  program  has 
been  written.  If  modularization  is  undertaken  at  this  point,  new 
interfaces  must  be  designed,  thus  possibly  increasing  the  program's 
complexity.  Also  the  entire  program  must  be  checked  to  determine  which 
parts  should  be  changed  and  which  parts  should  remain  unchanged,  as  a 
result  of  modularization.  Thus  once  reduced  to  code,  the  structural 
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complexity  of  a  program  is  essentially  fixed;  therefore,  it  is  imprac¬ 
tical  to  simplify  software  during  the  maintenance  phase. 

Peters  [5]  writes  that  much  of  the  difficulty  of  software  design 
stems  from  the  fact  that  the  problem  we  are  attempting  to  solve  is 
changing  while  we  are  solving  it.  Sane  of  these  changes  are  made  by  the 
designer  as  he  obtains  a  better  understanding  of  the  problem ,  and  other 
changes  come  from  the  user.  In  either  case  destabilization  of  the  pro¬ 
ject  and  reduction  in  quality  are  the  results.  This  situation  illus¬ 
trates  the  need  for  software  specifications  to  require  a  formal  change 
procedure .  Also,  Peters  warns  again s  the  fanaticism  of  some  promoters 
of  design  methods  and  techniques,  pointing  out  that  each  method  is 
directed  at  an  idealized  problem,  not  necessarily  the  one  to  be  solved. 
Although  there  may  be  same  merit  to  this  argument,  the  use  of  structured 
design  and  programming  techniques,  albeit  imperfect,  would  still  achieve 
consistency  of  documentation  and  a  disciplined  approach  to  design  and 
programming. 

Lientz  [6]  has  mentioned  that  a  principle  of  good  design  is  to 
design  with  enhancements  in  mind  because,  in  his  survey,  it  was  found 
that  most  (48  per  cent)  of  maintenance  activities  stenmed  from  enhance¬ 
ments.  This,  of  course,  has  a  direct  bearing  on  maintainability.  This 
is  a  point  well  made,  but  one  that  is  difficult  to  implement  in  practice 
due  to  the  difficulty  of  anticipating  future  operational  requirements 
during  the  design  stage.  Since  many  enhancements  are  the  result  of 
changing  data  requirements,  it  is  possible,  however,  to  avoid  certain 
practices  which  are  detrimental  to  maintenance,  such  as  imbedding  data 
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Cl 


td  standard 

should  call  for  independence  among  program  code,  data  and  data  bases. 

Heninger  and  colleagues  at  the  Naval  Research  Laboratory  have 
developed,  in  conjunction  with  the  A7E  aircraft  computer  system,  the 
concept  of  changes  which  are  likely  to  occur  and  those  which  are  not 
likely  to  occur  [7].  A  related  matter  is  the  identification  of  functions 
which  maintainers  would  like  to  modify  or  remove  easily,  should  the 
need  arise.  Easy  changes  should  correspond  to  the  most  likely  changes. 
These  considerations  evolve  into  a  design  principle  of  separating 
things  that  will  remain  the  same,  no  matter  what  changes  are  made  in 
the  rest  of  the  system,  from  those  things  that  will  be  affected  by  the 
changes.  System  documentation  should  only  specify  external  behavior 
without  implying  a  particular  implementation.  For  example,  programs 
should  not  have  to  be  changed  significantly  if  changes  are  made  to  the 
hardware.  In  other  words,  software  is  described  without  reference  to 
hardware.  Another  example  of  the  desired  immutability  of  software  is 
that  software  should  not  change  if  data  arrives  in  different  formats 
over  different  channels.  Thus  from  the  above,  a  software  specification 
and  standard  should  specify  that  software  functions  be  divided  into 
those  which  will  (likely)  change  and  those  which  will  not  (unlikely) 
change .  This  approach  will  have  the  desirable  result  of  causing  the 
designers  and  programmers  to  think  about  potential  change  requests 
early  in  system  development. 


10 


> 


B.  Specification  and  Documentation 


[ttjxgmgnta  for  Achieving  Good 


Maintainability 

One  of  the  best  approaches  for  designing  a  system  is  to  design  the 
documentation  first.  Considerable  emphasis  should  be  placed  on  docu¬ 
mentation  for  maintaining  software.  Hegland  suggests  determining 
documentation  requirements  at  the  beginning  of  a  project  [8].  Young  [9] 
lists  the  following  types  of  documentation  as  being  applicable  to  com¬ 
puter  software: 

Type  Project  Phase  Produced  In 


Functional  Requirements  Document 

Problem  Definition 

Data  Requirements  Document 

Problem  Definition 

System  and  Sub-System  Specifications 

System  Design 

Program  Specification 

System  Design 

Data  Base  Specification 

System  Design 

Test  Plan 

System  Design 

User  Manual 

Programming 

Operations  Manual 

Programming 

Program  Maintenance  Manual 

Programming 

Test  Analysis  Report 

Test 

To  the  above  should  be  added  an  Interfaces  Specification,  which  would 
be  produced  during  system  design.  A  software  specification  and  stein dard 
should  require  that  the  documentation  to  be  provided  on  a  project  be 
specified.  It  should  also  be  required  that  the  various  levels  of 
documentation  be  consistent  (e.g.,  sub-program  specifications  should  be 
consistent  with  the  associated  program  specification) . 


Since  good  maintainability  implies  good  readability  [lo],  the 


following  aids  to  readability  should  be  required  by  the  specification 
and  standard: 

-  Comments  per  line  or  related  lines  of  code. 

-  References  in  the  source  listings  to  software  specifications, 
and  approved  changes  and  test  plans. 

-  References  in  all  documents  to  related  documents. 

Jones  [11],  based  on  his  study  of  design  and  specification  techniques, 
concluded  it  is  impossible  to  conduct  a  large  software  project  without 
considerable  oral  canmunication  among  project  members;  it  is  not 
possible,  in  his  view,  to  conduct  a  project  mainly  on  the  basis  of 
written  documentation.  One  reason  for  this  situation  is  that  written 
documentation  may  become  so  bulky  that  it  ceases  to  be  useful.  As  a 
consequence,  according  to  Jones,  word  processing  costs  have  become  the 
second  largest  project  expense ,  behind  debugging  costs.  Despite  this 
result,  we  would  not  propose  a  reduction  in  project  documentation;  on 
the  contrary,  the  thoroughness  of  documentation  should  increase  in 
quality  and  quantity.  However,  the  possibility  of  inundating  a  project 
with  paperwork  does  suggest  the  need  for  a  specification  and  standard  to 
stipulate  oral  communication  in  the  form  of  design  reviews  and  walk¬ 
throughs  . 

Balzer  and  Goldman  [12 ]  mention  understandability  as  the  first 
criterion  for  judging  specifications.  Since  the  specifications  are  the 
basis  of  a  contract  between  contractor  and  customer,  they  must  be  clear 
and  unambiguous.  It  seems  appropriate .therefore,  for  a  software  standard 


tamer  (user  and  maintainer) .  These  authors  also  make  the  interesting 
point  that  the  specification  itself  must  be  testable.  That  is,  for  a 
specification  to  be  considered  valid,  it  is  necessary  to  demonstrate 
that  questions  (tests)  that  one  might  pose  concerning  proposed  system 
operations  can  be  answered  satisfactorily  by  the  specifications.  Thus, 
a  software  standard  should  require  that  the  exercise  of  "testing"  the 
specification  for  validity  be  a  part  of  the  specification  development 
process .  This  consideration  suggests  the  need  for  a  specification  model 
A  carefully  designed  standard  could  serve  as  the  model  for  writing  a 
specification.  Furthermore , the  authors  state  the  importance  of  making 
the  specification  independent  of  the  implementation.  Thus  a  specifica¬ 
tion  should  state  what  is  to  be  accomplished  and  not  how  it  is  to  be 
implemented.  Consequently,  another  objective  of  specification  develop¬ 
ment  which  should  be  stated  in  a  software  standard  is  the  objective  of 
achieving  independence  of  the  specifications  from  methods  of  implemen¬ 
tation  .  The  current  interest  in  specification  languages  as  a  means  of 
obtaining  consistency  and  under standability  fo  specifications  has 
implications  for  maintainability.  Balzer  and  Goldman  caution  against 
the  use  of  a  specification  language  which  optimizes  (reduces  the  resour¬ 
ces  required  to  execute  a  program)  a  system.  This  would  have  the 
undesirable  effects  of  reducing  understandability,  testability  and 


maintainability.  Thus  a  software  standard  should  stipulate  that 
optimization  techniques  are  not  to  be  employed  unless  their  use  is 


that  tricky  coding  techniques,  designed  to  achieve  optimization, 
frequently  lead  to  unmaintainable  software. 

Lastly,  these  authors  discuss  the  effects  of  specification  changes 
on  maintainability.  They  recommend  that  a  specification  be  so  designed 
that  changes  can  be  localized  and  not  have  an  effect  on  the  rest  of  the 
specification.  However,  there  are  instances  in  which  different  specifi¬ 
cations,  such  as  system  and  program  specifications,  or  different  parts 
of  a  given  specification,  must  be  related.  Where  these  relationships 
exist,  this  fact  must  be  visible  in  the  specification  so  that  changes  to 
related  specifications  can  be  made  when  a  change  is  made  in  a  given 
specification.  Thus,  with  regard  to  the  above  two  points,  a  software 
standard  should  stipulate  that  parts  of  a  specification  are  to  be  made 
as  independent  as  possible >  but,  where  relationships  must  exist,  these 
relationships  must  be  made  explicit  in  the  system  documentation . 

One  of  the  problems  with  system  development  is  lack  of  adherence  to 
stated  software  performance  objectives  over  the  software's  life  cycle. 

As  stated  by  McCall  [13],  little  attention  is  given  to  identifying  the 
qualitiies  that  the  software  should  exemplify  over  its  life  cycle.  What 
is  needed  is  a  clear  statement  of  performance  goals  in  the  user  require¬ 
ments  statement,  consistency  in  the  use  of  these  goals  in  subsequent 
stages  of  development  and  the  ability  to  trace  these  goals  forward, 
from  user  requirements  phase  to  maintenance  phase;  and  backward,  from 
maintenance  to  user  requirements.  The  achievement  of  these  objectives 
will  be  aided  if  a  software  specification  and  standard  requires  that 
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DeMillo  and  colleagues  [14]  believe  a  formal  demonstration  (test) 
that  a  program  is  consistent  with  its  specifications  has  value  only  if 


the  specifications  are  derived  independently  from  the  program.  In 
other  words,  the  specifications  should  be  derived  from  user  require¬ 
ments  and  not  from  the  program.  Then,  if  the  specification  reflects 
user  requirements  and  the  program  is  in  accordance  with  the  specifica¬ 
tion,  a  demonstration  of  the  program  would  be  meaningful .  if  the  above 
is  not  the  case,  the  following  situation  could  ensue:  A  program  fails, 
it  is  changed,  and  the  changes  are  based  on  faulty  specifications:  or 
the  specifications  are  changed,  and  those  changes  are  based  on  knowledge 
of  the  program  gained  through  the  failure.  In  either  case,  the  require¬ 
ment  of  using  independent  criteria  is  no  longer  met.  To  guard  against 
this  situation,  it  is  necessary  for  a  software  standard  to  specify 
that  software  design  or  performance  specifications  be  independently 
derived  from  user  requirements  and  not,  after-the-fact,  from  the  program 
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design.  Also,  as  pointed  out  by  Bamamoorthy  and  colleagues  [15]  the 
test  plan  should  be  independent  of  the  design  specification  but  depen¬ 
dent  upon  user  requirements.  If  this  is  not  the  case,  am  error  in 
translation  from  the  user  requirements  to  the  specification  will  not  be 
caught  if  the  test  plan  mirrors  the  specification.  The  error  will  be 
caught  if  the  test  plan  reflects  user  requirements.  In  other  words, 
softwaure  must  be  tested  against  what  the  software  was  intended  to  do 
and  not  against  what  it  is  doing. 

A  big  step  towards  error  reduction  in  software  would  result  from 
the  use  of  standard  and  certified  reuseable  modules  [11].  This  practice 


would  obviate  the  need  for  module  testing  of  reused  modules.  Only 
integration  testing  would  be  necessary  for  these  modules.  The  practical¬ 
ity  of  this  approach  is  less  apparent  in  embedded  computer  applications 
than  in  the  commercial  applications  due  to  the  nonstandard  nature  of 
the  former  (e.g.,  uniqueness  of  same  electronic  warfare  programs  relative 
to  accounts  payable  routines).  However,  certain  data  transformation  and 
algorithmic  routines  may  be  standard  (more  or  less) .  A  Software  standard 
should  require  the  use  of  reuseable  modules  wherever  appropriate  with 
standard  interfaces  between  modules. 

Design  problems  are  the  chief  source  of  programming  errors  and 
detecting  and  removing  errors  are  the  major  programming  costs  [ll]. 
Therefore ,  it  is  of  interest  to  identify  the  defect  removal  techniques 
which  work  best  with  a  given  design  approach.  Jones  [ll]  lists  four 
defect  removal  methods: 

-  Design  reviews . 

-  Testing . 

-  Correctness  proofs. 

-  Models  or  prototypes  for  verifying  designs. 

Two  major  categories  of  design  error  are:  (1)  leaving  out  needed  user 
functions,  and  (2)  putting  in  functions  that  users  don't  need.  Correct¬ 
ness  proofs  and  testing  are  not  guaranteed  to  find  these  errors.  However, 
methods  for  depicting  models  of  systems,  such  as  Hierarchy  Plus  Input- 
Process-Output  (HIPO)  charts  [16]  and  Composite  Design  [17],  in  combina¬ 
tion  with  design  reviews ,  do  provide  considerable  help  in  identifying 
missing,  unneeded,  redundant  and  ambiguous  functions  and  in  specifying 
how  the  functions  should  be  related  and  coupled  together.  These  methods 
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have  an  important  ancillary  benefit  by  providing  a  uniform  method  of 
system  documentation.  In  view  of  the  above,  it  is  important  to  not 
rely  on  one  or  two  methods,  such  as  correctness  proofs,  to  provide 
product  quality  assurance,  but  rather  to  employ  a  variety  of  methods 
where  the  choice  of  method  will  depend  upon  the  objective  of  the  test. 
Thus,  a  software  specification  and  standard  should  not  be  restrictive 
in  their  stipulation  of  product  quality  assurance  and  testing  methodolo¬ 
gies.  In  fact,  it  would  be  worthwhile  for  these  documents  to  enumerate 
the  various  methods.  (At  this  time,  methods  for  verifying  the  design 
against  user  requirements  coupled  with  formal  design  reviews  are  the 
best  methods  for  assuring  product  quality.)  In  addition,  consideration 
should  be  given  to  the  adoption  of  a  documentation  technique,  such  as 
HIPO,  as  a  standard  for  embedded  computer  development .  As  noted  by 
Pariseau  [18]  something  like  HIPO  would  have  improved  visibility  and 
documentation  on  the  Carrier  Based  Tactical  Support  Center  project.  In 
addition  to  the  use  of  a  documentation  standard,  one  of  the  most  impor¬ 
tant  improvements  that  could  be  made  in  the  documentation  specification 
is  the  use  of  examples,  perhaps  ones  from  past  projects.  Wtihout 
examples,  specifications  and  standards  appear  ethereal  to  the  designer 
and  programmer.  A  requirement  for  a  software  specification  and  standard 
to  show  examples  would  significantly  improve  their  usability.  A  related 
consideration  is  that  the  choice  of  documentation  technique  should  not 
be  left  to  the  contractor  (developer)  alone  to  decide. 

Schneidewind  [l9  ]  states  that  stress  or  saturation  testing  is  as 
applicable  to  software  as  to  hardware,  particularly  with  regard  to  pre¬ 


mission  testing,  when  it  is  essential  to  identify  marginal  hardware  and 
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software  (e.g.,  software  which  operates  satisfactorily  under  normal  load 
but  fails—  buffer  overrun— under  peak  load)  prior  to  the  mission.  One 
cannot  assume  that  the  software  remains  invariant  between  missions, 
because  program  changes  could  result  from  hardware  errors  (e.g.,  a 
transient  hardware  error  causes  changes  in  programs  residing  in  memory 
or  on  disks) ,  program  maintenance  activities  or  errors  in  revised  pro¬ 
grams  which  are  distributed  to  the  field. 

Other  test  procedures  which  will  improve  maintainability  are: 

-  Static  testing  (without  execution) . 

-  Dynamic  testing  (with  execution) . 

-  Regression  testing  (retest  of  all  modules  affected  by  a  software 
change) . 

-  Testing  specifically  for  the  response  of  the  system  to  unde¬ 
sirable  or  unexpected  events.  This  includes  stress  testing. 

-  Providing  for  the  repeatability  of  tests. 

-  Use  of  an  independent  quality  control  group. 

Thus  a  software  specification  and  standard  should  make  specific  reference 
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V.  EVALUATION  OF  WEAPONS  SPECIFICATION  WS  8506 


The  Fiscal  Year  1979  Command  and  Control  System  Maintenance  Agency 
(CCSMA)  System-Level  Software  Maintenance  Approach  and  Transition  Plan 
[20]  states  that  WS  8506  and  MIL-STD  1659  will  be  among  the  predominant 
factors  for  software  delivered  to  CCSMA  for  the  Trident  ships.  Further¬ 
more,  CCSMA  is  determining  what  software  logic  documentation  is  neces¬ 
sary  for  maintenance  and  where  the  voids  are .  The  transition  plan  from 
the  development  to  maintenance  phase  includes: 

-  Test  and  evaluation. 

-  Validation  of  documentation. 

-  Configuration  management. 

-  Introduction  of  system  into  the  fleet. 

-  Software  maintenance . 

-  Quality  assurance  and  audits. 

The  software  delivered  to  CCSMA  by  the  development  agency  will 
include : 

-  Computer  program  performance  specifications. 

-  Computer  program  design  specifications. 

-  Interface  design  specifications. 

-  Computer  program  operator's  manual. 

-  Test  specification  requirements . 

-  Test  procedures . 

Thus  it  is  of  interest  to  evaluate  WS  8506  in  this  section  and  MIL- 
STD  1679  in  the  next  section,  particularly  with  regard  to  their 
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suitability  for  maintenance .  This  evaluation  is  performed  by  applying 
the  underlined  criteria  developed  in  Section  IV.  Only  the  criteria  which 
are  applicable  to  WS  8506  are  utilized  in  this  section.  The  evaluation 
of  both  WS  8506  and  MIL-STD  1679  will  categorize  the  two  documents  as 
follows : 

-  No  coverage . 

-  Inadquate  coverage . 

-  Adequate  coverage . 

-  Excellent  coverage . 

In  addition  to  the  evaluation  against  the  criteria,  comments  will  be 
made  about  various  sections  of  the  documents,  where  appropriate. 
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TABLE  1 

WS  8506  EVALUATION 


Criterion 


WS  8506  Status 


A.  Design  Approaches. 
1 •  Modularity . 


2 .  Change  procedure . 

3.  Independence  of  code, 
data  and  data  base. 

4.  Separation  of  software 
functions  by  anticipated 
degree  of  change. 

B.  Specification  and  Documen¬ 
tation  Requirements. 

1 .  Specification  of 
documentation . 

2.  Consistent  documentation. 

3 .  Readability . 

4.  Design  reviews  and 
walkthroughs . 

5 .  Identification  of 
performance  require¬ 
ments  . 

C.  Test  Requirements  and 

Relationship  to  Specification 

Documentation  and  Design. 

1.  Statement  of  product 
assurance  methods. 

2.  Documentation  standard. 

3.  Use  of  documentation 
examples . 

4.  Regression  testing. 


Adequate  coverage  in  Sec.  6.2,  Com¬ 
puter  Program  Design  Specification 
Detailed  Requirements. 

No  coverage. 

No  coverage. 

Adequate  coverage  in  Sec.  5.2, 
Computer  Porgram  Performance  Speci¬ 
fication  Detailed  Requirements. 


Excellent  coverage  throughout. 


Excellent  coverage  throughout. 

No  coverage. 

No  coverage. 

Adequate  coverage  in  Sec .  5 . 2 . ,  Com¬ 
puter  Program  Performance  Specifica- 
Detailed  Requirements. 


Adequate  coverage  in  Sec.  11, 
Computer  Program  Test  plan. 

No  coverage. 

Excellent  coverage  throughout. 

No  coverage. 
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TABLE  1  (Continued) 


Criterion 


Undesirable  and  unexpected 
event  testing,  including 
stress  testing. 

Repeatability  of  tests. 

Independent  quality  control. 


WS  8506  Status 


No  coverage. 


Mo  coverage. 
No  coverage. 
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It  is  not  surprising  that  WS  8506  does  not  mention  the  "No  coverage" 
items  in  TABLE  1  since  these  items  pertain  to  technical  and  management 
advances  in  software  which  had  not  occurred  when  WS  8506  was  issued  in 
November  1971.  From  a  maintenance  standpoint  the  most  serious  voids 
are  lack  of  change  procedure;  documentation  standard,  such  as  HIPO;  and 
regression  testing.  The  lack  of  these  capabilities  impairs  traceability. 
The  use  of  Data  Item  Descriptions  (DID's)  to  cover  these  voids  might  be 
considered  if  WS  8506  is  to  continue  in  use  for  embedded  computer  systems, 
with  regard  to  Item  A.l,  modularity,  there  is  coverage  with  respect  to 
allocating  software  modules  by  function,  e.g.,  tracking,  but  no  discus¬ 
sion  of  structured  programming  since  this  technology  was  not  available 
at  the  time  of  publication. 

Other  Comments 

Comments  are  made  below  about  various  sections  of  WS  8506  indepen¬ 
dent  of  the  criteria  for  evaluation  which  were  used  in  the  foregoing. 

1.  Although  computer  performance  specification  is  a  commonly  used 
term  in  military  software,  it  could  be  inappropriate  on  two  counts: 

(1)  functional  specification  would  seem  more  appropriate  to  connote 
the  idea  of  implementing  user  functional  requirements,  and  (2)  the  word 
"performance"  is  frequently  used  to  denote  performance  measurement 

(CPU  and  memory  utilization) . 

2.  P.  3-2,  paragraph  3.13.  The  use  of  the  word  "operation"  seems 
inappropriate.  Better  terminology  might  be  task  or  process  (in  opera¬ 
ting  system  sense)  to  denote  activity  on  a  program  which,  in  turn,  is 
implemented  with  computer  instructions . 
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3.  P.  4-1.  Other  types  of  documentation,  such  as  decision  tables, 
data  flow  diagrams  and  directed  graphs,  could  be  mentioned. 

4.  P.  5-1,  paragraph  1.2.  A  matrix  would  be  useful  for  depicting 
function  dependencies. 

5.  P.  5-2,  section  3.  The  distinction  among  functional,  operational 
and  performance  requirements  is  not  clear;  some  seem  to  be  subsets  of  the 
others . 

6.  P.  5-3.  The  various  types  of  documentation  should  be  related  in 
a  heirarchy,  possibly  with  HIPO  charts. 

7 .  P.  5-7.  The  use  of  state  diagrams  could  be  useful  for  showing 
processing  states  and  state  transitions. 

Paragraph  3.3.N.2.  Processing  should  be  related  to  the  use  of 
data  bases. 

8.  P.  5-10,  paragraph  3.4.  Discussion  of  data  base;  system  growth; 
recovery;  and  capacity,  storage  and  time  requirements  is  good. 

9.  P,  5-11,  paragraph  4,1.  All  tests  should  be  documented. 

Paragraph  4.2.  Discussion  of  use  of  tools  and  tolerances  is  good. 

Paragraph  4.3.  Discussion  of  acceptance  testing  and  criteria  is 

good.  Should  specify  which  items  can  be  tested  by  simulation. 

10 .  P.  6-1.  Technique  for  specifying  programs  is  good. 

Section  3.1.  Allocation  of  functions  to  programs  is  good. 

11.  P.  6-2.  Identification  of  tasks  is  good. 

Paragraph  3.2.  Functional  description  is  good. 

Paragraph  3.3.  Discussion  of  allocation  of  storage  and  processing 
time,  sequencing  requirements  and  equipment  constraints  is  good.  Figure 
6-2  is  helpful. 


24 


Si 


Paragraph  3.4.  Mention  of  flow,  both  control  and  data  is 
particularly  good. 

12.  P.  6-7.  paragraph  3.4.1.  Discussion  of  interrupts  is  good. 

Paragraph  3.4.2.  Discussion  of  subprogram  reference  is  good. 

Paragraph  3.5.  Discussion  of  use  of  monitor,  loader,  etc.  is 

good;  even  talks  about  configuration  management.  Mention  of  labeling 
conventions  is  good. 

13.  P.  6-8,  section  4.  This  Quality  Assurance  section  should  be 
expanded  to  include  discussion  of  independent  quality  control  group  and 
organization  of  the  quality  control  function. 

14.  P.  7-1.  Considering  system  subroutines  in  the  same  light  as 
subprograms  is  good.  The  question  arises  as  to  the  appropriateness 

of  program  level  documentation  for  operational  personnel;  the  programming 
language  orientation  seems  inappropriate.  WS  8506  should  be  updated  to 
reflect  the  use  of  stacks,  reentrant  code,  separation  of  code  and  data  and 
use  of  concurrent  processes. 

15.  P.  7-3,  paragraph  3.3.  Discussion  of  data  base  usage  is  good. 

16.  P.  7-5,  paragraph  3.3.1.  Discussion  of  table  documentation  is 
good.  The  discussion  of  bit  layouts  and  flags  is  incompatible  with 
today's  emphasis  on  use  of  higher  order  languages  (ROD.  This  section 
should  be  revised  to  reflect  greater  HOL  usage.  Perhaps  the  specifica¬ 
tion  could  be  divided  into  different  parts  depending  upon  whether  assem¬ 
bly  language  or  HOL  is  used. 

17.  P.  7-6,  paragraph  3.4.  Discussion  of  I/O  formats  is  good;  disc 
formats  should  be  added. 
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and  referencing  is  good.  Might  consider  use  of  decision  tables  for 
stating  conditions  of  subroutine  use. 


19.  P.  7-9.  Diagram  of  subprogram  relationships  is  good. 

20.  P.  8-1,  section  8.1.  Good- tie  back  to  other  parts  of  specifica¬ 
tion. 

Section  8.2.  Mention  of  options  good. 

21.  P.  8-2,  paragraph  3.1.  Mention  of  table  indexing  and  initial 
condition  procedures  is  good. 

22.  A  list  of  acronyms  would  be  helpful. 


Summary 

WS  8506  is  considered  to  be  well  thought  out,  comprehensive  and  a  very 
good  specification  for  the  documentation  of  program  development ,  particu¬ 
larly  in  view  of  the  early  publication  date  of  this  document.  The 
strategy  of  making  each  level  of  documentation  responsive  to  the  next 
upper  level  (subprogram  design  under  program  design) ,  represents  fore¬ 
sight  in  the  use  of  top-down  design  prior  to  the  time  this  term  was  in 
vogue.  It  cannot  be  faulted  for  not  including  programming  technologies 
which  had  not  been  developed  at  the  time  of  its  publication .  However, 
the  document  is  less  useful  for  maintenance  purposes  for  the  reasons 
previously  given. 


MIL- STD  1679  is  of  great  importance  today  because  it  is  being  con¬ 
sidered  for  recansnendation  for  adoption  as  a  Department  of  Defense 
software  standard,  along  with  MIL-S-52779  as  a  Department  of  Defense 
software  specification,  by  the  Joint  Logistics  Commanders,  Joint  Policy 
Coordinating  Group  on  Computer  Resource  Management  [21], 

The  evaluation  of  MIL- STD  1679  against  the  applicable  criteria 
developed  in  Section  IV  follows. 


TABLE  2 

MIL- STD  1679  EVALUATION 


Criterion 


MIL- STD  1679  Status 


Design  Approaches 

1 .  Modularity . 

2.  Change  Procedure. 

3.  Independence  of  code,  data 
and  data  base. 

4.  Separation  of  software 
functions  by  anticipated 
degree  of  change. 

Specification  And  Documentation 

Requirements 

1.  Specification  of  documen¬ 
tation  . 

2.  Consistent  documentation . 

3 .  Readability . 

4.  Design  reviews  and  walk¬ 
throughs  . 

5 .  Under standability . 

6.  Specification  testing. 


Excellent  coverage  in  Sec.  5.2, 
Program  Design  Requirements. 

Excellent  coverage  in  Sec.  5.11.2, 
Configuration  Control. 

Adequate  coverage  in  Sec.  5.4.1, 
Symbolic  Parameterization. 

Inadequate  coverage  in  Sec.  5. 1.2. 7, 
Adaptive  Parameters. 


Excellent  coverage.  Examples : 

Sec.  5.6.1,  Supporting  Information 
for  Program  Performance  Requirements; 
Sec.  5. 1.2. 3.,  Applicable  Documenta¬ 
tion  for  Program  Performance  Require¬ 
ments;  and  Sec.  6.1,  Contract  Data 
Requirements . 

Adequate  coverage  in  Sec.  4.5, 
Configuration  Management. 

Adequate  coverage  in  Sec.  5.3.10, 
Indentation;  and  Sec.  5. 4. 4. 2, 

Comment  Statements. 

Excellent  coverage  in  Sec.  5. 9. 1.3, 
Design  Reviews. 

No  coverage. 

Adequate  coverage  in  Sec.  5.12.3.3, 
Documentation  Reviews. 

No  coverage . 


7 .  Implementation  indepen¬ 
dence  of  specification. 


TABLE  2  (Continued) 


Criterion 


MIL- STD  1679  Status 


8.  Non-use  of  optimization  Adequate  coverage  in  Sec.  4.2, 

techniques.  Design  Requirements,  but  inade¬ 

quate  coverage  in  Sec.  S.4.S.1, 
Execution  Efficiency. 

9.  Independence  of  specifics-  No  coverage, 

tion  puts. 

10.  Identification  of  performance  Excellent  coverage  in  Sec.  5. 9. 1.4, 
requirements.  Program  Design. 


C.  Test  Requirements  and  Relationship 
to  Specification  Documentation  and 
Design. 

1.  Independence  of  performance 
specifications  and  software 
design  from  program  design. 

2.  Use  of  reuseable  modules. 

3.  Statement  of  product 
assurance  methods. 


4.  Documentation  standard. 


5.  Use  of  documentation 
examples . 

6.  Regression  testing. 

7.  Undesirable  and  unexpected 
event  testing  including 
stress  testing. 

8.  Repeatability  of  tests. 

9.  Independent  quality  control. 


No  coverage . 


No  coverage. 

Excellent  coverage  in  Sec.  5.9, 
Quality  Assurance;  and  Sec.  5.10, 
Program  Acceptance. 

Inadequate  coverage.  Only  struc¬ 
tured  programming  (Fig.  1)  conven¬ 
tions  are  used. 

Inadequate  coverage.  Only  one 
Example  in  Fig.  1,  Control  Struc¬ 
tures. 

No  coverage . 

Excellent  coverage  in  Sec.  5.10.2.8. 
Software  Quality  Test  Stress  Testing. 


No  coverage. 

Excellent  coverage  in  Sec.  5. 9. 1.1, 
Reporting  Level. 
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MIL-STD  1679  includes  developments  in  programming  technology  and 
management,  such  as  structured  programming  and  walkthroughs  which  have 
occurred  since  the  advent  of  WS  8506.  The  major  deficiencies  are  that 
no  mention  is  made  of  the  need  to  independently  derive  performance  re¬ 
quirements  from  user  requirements  and  the  need  to  achieve  relative  inde¬ 
pendence  of  the  various  parts  of  the  system  and  programs.  From  a 
maintenance  standpoint,  the  former  capability  is  important  in  order  to 
achieve  traceability.  The  latter  capability  is  important  in  order  to 
localize  the  effects  of  maintenance  changes.  Also,  the  lack  of  a  regres¬ 
sion  testing  specification  is  a  hindrance  for  maintenance  because  this 
type  of  testing  should  be  performed  subsequent  to  maintenance  modifications. 

Other  Cawnents 

Comments  are  made  below  about  various  sections  of  MIL- STD  1679 
independent  of  the  criteria  for  evaluation  which  were  used  in  the  fore¬ 
going  . 

1.  P.  1,  section  1.2.  It  is  good  to  have  included  firmware  in  the 
standard. 

2.  P.  6,  section  4  2.  It  is  good  to  mention  that  design  complexity 
and  system  interdependencies  should  be  minimized. 

Section  4.3.  It  is  good  to  stipulate  use  of  a  HOL. 

3.  Section  4.5.  It  is  good  to  specify  configuration  management 
for  correlating  documentation  with  the  program  for  maintenance  purposes. 

4.  P.  7,  section  5.1.  It  seems  inappropriate  for  the  contractor, 
alone,  to  determine  program  performance  requirements. 
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5.  P.  8,  section  5. 1.2. 5. c.  and  d.  Same  comment  as  #4.  with  regard 
to  intersystem  interface  and  function  description. 

6.  P.  9,  section  S.2.  Same  comment  as  #4.  with  regard  to  program 
architecture . 

Section  5.2.2.3a.  Predicting  the  rate  of  interrupt  occurrences 
would  be  very  difficult. 

7.  P.  10,  section  5.2.3.  Seems  to  be  repetitious  of  Section  5. 1.2. 5. c. 

Section  5.3.1.  Use  of  words  "compile-time  system”  is  not  clear. 
Section  5.3.8.  Allowing  compiler  to  generate  backward  jumps 

seems  inconsistent  with  objective  of  this  section. 

8.  P.  14,  section  5.4.3.  Numerical  conventions  should  be  established 
by  the  government. 

9.  P.  15,  section  5.4.6.  Although  flow  charts  may  not  be  a  neces¬ 
sary  part  of  documentation,  some  sort  of  graphic  documentation,  such 

as  block  diagrams  or  control/data  flow  graphs,  should  be  required  in 
addition  to  program  listings. 

Section  5.5.2.  Same  comment  as  #4  with  regard  to  resource  management. 
Section  5.5.3.  Repetitious  of  section  4.3. 

10.  P.  16,  section  5.5.4.  Patches  should  be  disallowed. 

Section  5.7.  Same  comment  as  #4.  with  regard  to  program 

operation . 

11.  P.  17,  section  5.8.1.  Should  specify  the  test  data  to  be  used 
with  module  testing. 

Section  5.8.2.  Implies  bottom-up  testing.  Standard  should 
not  preclude  the  use  of  top-down  testing. 
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12.  P.  18,  section  5.8.5.  Sams  comment  as  #4.  with  regard  to  soft¬ 
ware  trouble  reporting. 

Section  5.8.5.1.b.  Add  the  words  "and  the  documentation  is 
defective”  to  the  end  of  sentence. 

13.  P.  19,  section  5.8.5.1.d.  This  could  be  classified  as  a  specifi¬ 
cation  trouble. 

Section  5. 9. 1.2.  Participation  in  audits  should  start  earlier 
with  user  requirements  definition. 

14.  P.  20,  section  5. 9. 1.6.  Same  comment  as  #4.  with  regard  to 
conduct  of  tests. 

Section  5.10.  Program  acceptance  seems  to  be  based  on  some 
strange  criteria,  such  as  unresolved  software  and  documentation  errors 
and  extent  of  patches. 

Section  5.10.2.  Should  specify  how  test  time  is  to  be  determined. 
The  use  of  the  words  "reasonably  free"  should  be  defined.  It  seems  that 
if  the  software  is  stressed  beyond  its  design  capacity,  the  result  will 
be  catastrophic. 

15.  P.  21,  section  5.10.2.7.  It  seems  this  section  could  be  more 
appropriately  named  performance  test. 

16.  P.  22,  section  5.10.2.8.  Perhaps  what  is  meant  here  is  to  cut 
power  rather  than  secure  power. 

Section  5.10.3.  and  section  5.10.3.2.  Use  of  patch  limits  as  a  soft¬ 
ware  quality  test  limit  is  poor. 

17.  P.  23,  section  5.11.  Configuration  management  should  include 


hardware 


Software  Configuration  Control  Boards. 


19.  P.  26,  section  5. 12. 3. 3. b.  It  would  seem  more  appropriate  for 
the  government  to  schedule  documentation  reviews. 

Summary 

Overall,  MIL- STD  1679  appears  to  be  a  good  software  standard  for 
embedded  computer  software,  primarily  because  it  addresses  areas  not 
emphasized  in  previous  standards  (e.g.,  change  control).  It  seems  to 
be  the  best  standard  available ,  both  for  development  and  maintenance, 
although  it  should  be  noted  that  its  companion  document  for  DOD  adoption, 
MIL-S-52779,  is  intended  only  for  software  acquisition  and  is  encouraged 
but  is  not  mandatory  (as  of  this  writing)  for  use  on  maintenance  contracts. 
There  are  two  disturbing  aspects  of  MIL- STD  1679,  one  of  which  could  have 
a  serious  effect  on  maintenance;  this  is  the  allowance  of  patches  to 
the  extent  of  the  limits  given  in  Section  5.10.3.2.  The  other  is  the 
seemingly  excessive  control  and  responsibility  which  is  delegated  to 
the  contractor. 
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author  and  do  not  necessarily  reflect  the  opinions  of  the  Naval  Post¬ 
graduate  School,  Department  of  the  Navy,  or  Department  of  Defense. 
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